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1.0  INTRODUCTION 

The  recent  spread  of  the  use  of  Unmanned  Air  Vehicles  (UAVs)  for  military  purpose  over  the  past  two 
decades  have  led  the  industry  to  develop  various  payloads,  matching  the  operational  needs  for  the  various 
missions  committed  to  the  UAV  systems.  In  this  context,  SAGEM  has  positioned  as  a  main  actor  in 
Europe  in  the  field  of  UAV: 

•  European  leader  for  the  tactical  UAV  systems  (SPERWER). 

•  Major  sub-contractor  for  observation  payloads  dedicated  to  medium  altitude  long  endurance  UAV. 

•  Proposition  of  “Over-the-Hill”  mini-UAV  concepts  in  the  frame  of  FELIN. 

This  position  has  notably  been  acquired  thanks  to  the  long  term  know-how  of  SAGEM  in  several  domains, 
particularly  critical  in  the  design  of  a  UAV  system: 

•  European  leader  of  optronic  systems. 

•  European  N°l,  World  n°3  for  Inertial  Navigation  Systems. 

•  Expert  in  Global  Positioning  Systems,  RF  communication  and  Data  Links. 

In  the  frame  of  these  developments  in  the  field  of  UAV  systems,  a  study  has  been  dedicated  to  the 
definition  of  new  observation  payloads,  in  order  to  anticipate  the  various  needs  for  the  future  UAV 
systems.  This  study  put  a  particular  focus  on  the  possible  commonalties  between  the  different  payloads 
responding  to  the  various  needs,  and  on  the  synergies  which  could  be  obtained  between  the  different  sub¬ 
elements  of  the  observation  payload,  and  with  the  other  payloads  embedded  on  the  UAV. 

The  next  sections  of  the  paper  will  cover  the  following  topics: 

•  General  presentation  of  the  usual  architecture  of  the  UAV  observation  function. 

•  Functional  evolution  of  the  SAGEM  airborne  observation  payloads. 

•  Specificity  of  the  UAV  context,  with  respect  to  other  airborne  platforms. 

•  Presentation  of  different  data  and  image  fusion  techniques  which  can  be  implemented  in  the  UAV 
observation  payload,  and  discussion  on  their  functional  interest. 

•  Proposition  of  an  UAV  observation  function  architecture  optimised  for  data  and  image  fusion. 
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2.0  ARCHITECTURE  OF  THE  UAV  OBSERVATION  FUNCTION 

The  observation  (or  imagery)  function  of  a  UAV  system  has  long  been  the  main  function  for  the  final  user, 
and  remains  one  of  the  primary  functions  even  since  the  use  of  UAVs  for  other  applications,  as  signal 
jamming,  target  designation,  or  weapons  carrying.  This  observation  function  is  ensured  by  several 
elements  of  the  UAV  system,  constituting  the  UAV  Imagery  Chain: 

•  On  the  UAV: 

•  The  UAV  observation  payload  (UAV-OP)  itself,  constituted  of  various  sensors  and 
electronics  depending  of  the  operational  needs,  integrated  in  a  mechanical  assembly  allowing 
possibly  the  orientation  of  the  line-Of-Sight  (LOS). 

•  The  UAV  Video  &  Data  Link  Payload  (UAV-VDLP),  which  compress  (if  necessary)  the 
motion  imagery  and  the  data  coming  from  the  UAV-OP,  and  send  them  to  the  ground  station, 
possibly  through  the  medium  of  a  satellite  or  another  UAV. 

•  The  UAV  Management  &  Control  Unit  (UAV-MCU),  which  receives  the  operator  commands 
through  the  UAV-VDLP,  transmits  or  translates  them  in  lower  level  commands  to  the  various 
payloads  embedded  in  the  UAV. 

•  On  the  ground  station  (GS): 

•  The  GS  Video  and  Data  Link  Module  (GS-VDLM)  is  the  symmetrical  element  to  the  UAV- 
VDLP,  in  charge  of  imagery  and  data  coding  /  decoding. 

•  The  GS  Video  &  Data  Processing  Unit  (GS-VDPU)  is  in  charge  of  additional  processing  on 
the  imagery  and  data  sent  by  the  UAV,  and  of  processing  of  data  and  commands  collected 
through  the  Man-Machine-Interface. 

•  The  GS  Man-Machine-Interface  (GS-MMI)  ensures  the  collection  of  commands  and 
parameters  necessary  to  the  mission  and  the  presentation  of  imagery  and  data  to  the  UAV 
system  operators. 

An  example  of  UAV  system  architecture  is  given  in  the  figure  below: 
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Figure  1:  An  example  of  UAV  system  architecture. 


Each  element  of  the  imagery  chain  play  a  significant  role  in  the  final  performance  of  the  function,  but  their 
respective  intrinsic  weight  in  the  system  performance  varies  according  to  the  type  of  UAV  system. 


3.0  EVOLUTION  OF  THE  AIRBORNE  OBSERVATION  PAYLOADS 

During  the  last  three  decades,  the  optronic  devices  used  for  observation  purpose  have  known  a  continuous 
progress,  on  various  airborne  platforms  (helicopters,  planes,  UAV).  This  constant  evolution  has  been  made 
possible  by  major  technical  improvements  on  different  sub-elements  of  observation  payloads: 

•  Development  of  compact  EO/IR  sensors, 

•  Increased  capabilities  of  embedded  electronics  (sensor  management,  signal  processing), 

•  Robust  LOS  stabilisation, 

•  Lightweight  structure  for  platforms . . . 
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The  two  major  trends  coming  out  of  these  evolutions  are: 

•  The  increase  of  the  number  and  diversity  of  sensors  embedded  in  airborne  observation  payloads, 
due  to  compactness  improvement. 

•  The  enlargement  of  the  functional  area  covered  by  the  observation  payload  sub-system,  gained  on 
the  functional  responsibility  of  the  Host  System. 

3.1  Embedded  Sensors  on  Airborne  OP 

Early  observation  payloads  featured  an  only  sensor,  a  daylight  camera  or  a  thermal  imager,  which  allowed 
to  cover  day  &  night  operations.  Depending  on  the  maximum  weight  devoted  to  the  payload,  these  two 
types  of  sensors  could  be  embedded  simultaneously  on  the  same  carrier. 

A  typical  daylight  camera  used  in  airborne  OP  application  is  based  on  a  CCD  detector  with  TV  resolution, 
associated  to  a  fixed  Field  of  View  (FOV)  optics,  or  to  a  zoom  lens.  Main  progress  in  the  field  of  daylight 
cameras  consisted  in  the  improvement  of  CCD  sensitivity  and  the  increase  of  the  optical  zoom  range. 

The  thermal  imagers  or  infrared  cameras  used  in  airborne  OP  have  followed  the  successive  steps  of  this 
technology  during  the  past  three  decades: 

•  First  thermal  imagers  used  for  airborne  observation  where  line  scanners  used  in  conjunction  with 
whisk  broom  technique:  a  linear  detector  mounted  on  a  moving  arm  is  quickly  swept  along  a 
direction  perpendicular  to  the  carrier  movement  to  generate  a  panoramic  image  of  the  area  below 
the  airborne  carrier. 

•  In  2nd  Generation  thermal  imagers,  the  scanning  is  ensured  by  an  opto-mechanical  device 
integrated  in  the  camera,  allowing  a  drastic  reduction  of  the  sensor  volume,  and  the  production  of 
thermal  imagery  at  video  frame  rate. 

•  3rd  Generation  thermal  imagers  are  characterised  by  their  Focal  Plane  Array  (FPA)  detector, 
producing  imagery  without  scanning  device. 

•  Last  generation  thermal  imagers  are  based  on  detectors  which  do  not  require  any  cryogenic 
module  to  produce  high  sensitivity  thermal  imagery,  contrary  to  previous  generation  IR  cameras: 
this  improvement  still  further  reduce  the  total  weight  and  volume  of  the  detector  and  electronics 
parts. 

Continuous  progress  in  miniaturisation  of  sensors  and  electronics  made  possible  the  integration  of  these 
imaging  sensors  in  smaller  and  lighter  payloads,  and  /  or  the  integration  of  bigger  optics,  allowing  a 
significant  increase  of  the  optronic  performances  of  the  payload  (Detection,  Recognition,  Identification  or 
DRI).  Another  possibility  offered  by  this  size  reduction  was  to  integrate  in  the  same  volume  a  second 
sensor  of  the  same  type,  with  different  optics  to  extend  the  zoom  range  in  this  spectral  band:  for  example, 
a  TV  camera  with  very  high  magnification  optics  (named  spotter)  can  be  used  next  to  a  classical  TV 
camera  with  a  zoom  to  provide  extra  identification  capabilities. 

In  parallel  of  this  continuous  evolution,  the  next  step  was  the  integration  of  laser  sensors,  bringing 
additional  functionalities  to  the  observation  function:  Laser  Range  Finder  provides  the  operator  with 
precise  measurements  of  the  observed  target  distance,  Laser  Pointer  allows  to  precisely  indicate  a  zone  or 
a  target  to  another  observer  equipped  with  night  vision  sensor...  Another  laser  equipment  embedded  on 
airborne  OP  is  the  laser  designator  or  illuminator:  this  device  provides  the  airborne  system  with  the 
capability  to  designate  a  target  to  a  laser  guided  ammunition,  often  combined  with  a  range-finding 
functionality. 

Today,  a  state-of-the-art  Sensor  Pack  (SP)  may  include  up  to  six  different  sensors,  operating  in  five  or 
more  spectral  band,  for  example: 
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•  B&W  or  Color  High  Definition  TV  (HDTV)camera  (visible  light)  with  a  zoom  lens, 

•  Thermal  imager,  operating  in  3  -  5  pm  or  8  -  12  pm,  with  switchable  (FOV)  or  zoom  lens, 

•  Spotter,  operating  in  the  visible  or  in  the  Near-InfraRed  (NIR)  spectrum  (0.78  -  1  pm),  with  a 
fixed  high  magnification  lens  (typically  <0.5°), 

•  An  ey esafe  Laser  Range  Finder,  with  a  emission  wavelength  around  1.5  pm, 

•  A  Laser  Pointer,  compatible  with  night  vision  goggles,  sensitive  in  the  NIR  spectrum, 

•  A  Laser  Designator,  whose  emission  wavelength  is  1.06  pm. 

3.2  Functionalities  Covered  by  OP 

The  “functional  area”  covered  by  the  airborne  OP  itself  among  all  the  functionalities  of  the  airborne 
system  has  kept  extending  between  the  successive  improvements  of  the  system: 

1)  The  functions  covered  by  the  early  observation  payloads  were  reduced  to  the  functionalities  of  the 
only  sensor  itself.  The  still  or  motion  imagery  issued  by  the  sensor  was  directly  sent  to  the  host  system 
in  a  raw  format,  and  no  capability  of  sensor  orientation  was  available. 

2)  The  next  step  of  OP  evolution  was  the  association  of  a  sensor  with  an  orientation  mechanism:  the 
Sensor  Pack,  constituted  of  the  sensor  and  a  rate  gyroscope,  is  mounted  inside  a  Gyrostabilized 
Platform  (GP)  or  gimbal,  which  provide  2-axis  angular  orientation  of  the  imaging  sensor  Line-Of- 
Sight  (LOS).  At  this  stage,  the  SP  and  the  GP  were  considered  as  two  distinct  sub-systems,  piloted 
independently  by  the  Host  System  to  perform  low-level  functions  (Examples  of  low-level  observation 
functionalities:  FOV  change,  Gain  and  Offset  settings,  Electronic  zoom  activation,  LOS  Rate  piloting, 
LOS  orientation  in  the  carrier  reference. . .). 

3)  A  further  improvement  of  the  OP  was  allowed  by  progress  in  the  field  of  gyrostabilised  platforms:  in 
the  same  external  volume  as  the  previous  generation  gimbals,  it  was  now  possible  to  build  a  4-axis 
gyrostabilized  platform,  with  a  bigger  internal  volume  dedicated  to  the  sensors  integration.  The  4-axis 
gyrostabilisation  provided  a  high  quality  stabilisation,  necessary  to  perform  long  range  observation  on 
airborne  carriers.  The  extended  internal  volume  allowed  the  integration  of  several  sensors.  At  this 
stage,  the  sensor  pack  and  the  GP  was  considered  as  two  sub-assemblies  of  the  observation  payload 
sub-system,  but  they  were  still  commanded  through  two  separate  command  lines  with  low-level 
commands.  But,  in  addition  of  functionalities  already  covered,  the  OP  took  part  in  high-level 
observation  functionalities,  managed  by  the  airborne  carrier.  Examples  of  these  higher-level 
functionalities  are  given  hereunder: 

•  Tracking:  Automatic  orientation  of  the  LOS  on  a  target  detected  in  the  sensor  imagery, 

•  Designation  or  Geo-Tracking:  Automatic  orientation  of  the  LOS  towards  a  point  defined  in 
the  terrestrial  reference,  by  compensation  of  the  angular  (Roll,  Pitch,  Yaw)  and  linear 
(Vx,Vy,Vz)  motion  of  the  carrier, 

•  Geo-Scanning:  Automatic  LOS  piloting  ensuring  full  coverage  on  an  area  of  interest  on  the 
ground,  whatever  the  movements  of  the  carrier, 

•  Image  enrichment,  by  the  real-time  incrustation  of  symbology  and  alphanumerical  characters. 

4)  In  the  next  step  of  this  evolution,  the  management  of  above-mentioned  higher  level  observation 
functionalities  is  transferred  from  the  Host  System  management  system  to  the  OP:  In  airborne  mission 
system  connecting  numerous  and  complex  payloads,  the  reduction  of  the  exchange  between  carrier 
management  system  and  the  payloads  to  high  level  commands  significantly  reduces  the  amount  of 
data  circulating  on  the  mission  bus. 
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This  increase  of  computational  work  for  the  OP  was  materialised  by  the  development  of  specific 
electronics  and  software,  able  to  translate  the  Host  System  high  level  commands  and  dispatch  them  to  the 
various  sensors  aboard  the  payload.  These  electronic  boards  can  be  integrated  in  a  separate  unit  or  on  the 
sensor  pack,  depending  on  the  space  available  inside  the  platform. 

5)  The  last  trends  of  evolution  of  the  role  of  the  OP  in  the  airborne  system  are  toward  a  greater 
integration  and  modularity  of  the  OP  and  toward  an  active  implication  of  the  OP  in  functionalities  at 
airborne  system  level.  This  implication  may  vary  with  the  type  of  airborne  system:  On  an  Army 
helicopter,  the  OP  may  help  the  pilot  for  collision  avoidance  during  low  level  flight;  on  a  tactical 
UAV,  the  Laser  Range  Finding  will  be  used  to  confirm  the  elevation  of  a  numerical  model;  on  mini  or 
micro  UAV,  imagery  will  be  used  to  assist  the  navigation  and  piloting  functions. . . 


4.0  SPECIFICITY  OF  THE  UAV  CONTEXT 

The  use  of  an  observation  payload  in  an  UAV  system  presents  some  specificity  when  compared  with  the 
integration  on  other  airborne  carriers  (helicopters,  patrol  planes. . .). 

The  main  characteristics  of  OP  dedicated  to  UAV  are  of  course  linked  to  the  fact  that  the  carrier  is 
inhabited: 

•  First,  the  information  gathered  by  the  payload  is  not  exploited  on  the  carrier  but  on  the  ground 
after  transmission,  which  makes  necessary  a  specific  processing  and/or  compression  before 
transmission.  On  the  other  hand,  this  specificity  allows  to  transfer  into  the  GS  a  part  of  signal 
processing  usually  done  inside  the  OP,  and  even  perform  in  the  GS  additional  processing  which 
could  not  be  done  on  a  carrier,  because  of  the  limitation  on  the  weight  of  the  OP. 

•  Second,  the  delay  induced  by  the  transmission  of  Ground-to-UAV  commands  and  UAV-to- 
Ground  data  leads  the  UAV  system  designer  to  give  a  high  level  of  autonomy  and  automation  to 
the  UAV  and  to  its  sub-systems,  and  among  them  the  OP:  this  issue  is  particularly  critical  to 
ensure  continuous  operation  when  the  Data  Link  is  interrupted.  In  addition,  the  autonomy  of  the 
UAV  relies  on  various  payloads,  which  provides  the  UAV-MCU  the  necessary  data  to  perform 
automatic  piloting,  navigation...  This  connection  of  the  observation  payload  to  a  rich  and  precise 
data  network  is  another  particular  to  the  UAV  context. 

Another  trend  of  UAV  payloads’  needs  is  a  constant  increase  of  the  required  performances,  exceeding  the 
requirements  for  other  airborne  carriers.  Linked  to  the  diversification  of  UAV,  the  required  improvement 
may  concern  different  characteristics  and  performances  in  function  of  the  UAV  type. 

•  Optronic  performances:  with  the  development  of  medium  and  high  altitude  UAV,  a  constant 
improvement  of  DRI  range  of  imaging  sensors  is  required,  in  order  to  keep  at  least  identical 
operational  capability  at  increased  altitude  and  distance. 

•  Geo-Localisation:  In  relation  with  the  increase  of  UAVs’  altitude,  the  level  of  geo-localisation 
precision  required  demands  both  highly  accurate  LOS  orientation  and  high  quality  imagery. 

•  Miniaturisation:  the  expression  of  a  increasing  need  for  UAV  systems  usable  at  platoon  level,  for 
short  distance  (“Over-The-Hill”)  reconnaissance  missions,  or  specifically  developed  for  military 
operations  in  urban  terrain,  put  a  strong  pressure  on  the  miniaturisation  of  optronic  sensors,  to 
provide  the  soldier  with  an  extended  observation  capability,  compatible  with  its  operational 
requirements:  night  and  day  imagery  allowing  recognition  at  firing  distance,  transportability, 
robustness... 

The  last  characteristic  specific  to  UAV  needs  is  the  necessary  effort  on  interfaces  standardisation. 
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1)  A  major  trend  in  the  field  of  UAV  systems  is  the  growing  diversity  of  applications  in  which  they  are 
used,  resulting  in  the  development  of  different  UAV  systems  responding  to  these  specific  needs,  and 
thus  in  the  development  of  various  payloads. 


Figure  2:  Diversification  of  UAV  types  on  examples  of 
SAGEM  present  UAVs,  contributions  and  concepts. 


2)  On  piloted  air  vehicles,  imagery  and  data  produced  by  the  OP  are  mainly  used  on  board  in  real-time,  at 
the  time  they  are  displayed  on  the  operator  screen.  This  information  may  be  used  later  if  it  has  been 
recorded  or  transmitted,  but  this  use  remains  a  minor  functionality  with  respect  to  the  primary  mission 
of  observation  (at  least  on  existing  systems).  The  OP  only  has  to  be  compatible  with  the  vehicle 
display  and  control  unit  (if  not  a  sub-system  of  the  OP  itself)  to  be  used  on  a  piloted  aircraft. 

On  the  other  side,  in  UAV  systems,  the  link  between  the  OP  and  the  operator  relies  on  several  other  sub¬ 
systems  (see  §2.1)  which  have  to  be  compatible  to  each  other  in  terms  of  functional  and  data  interfaces. 
With  the  diversification  of  payloads  and  air  vehicles  specific  to  different  missions  and  conditions,  it 
appears  as  a  necessity  to  standardise  interfaces  between  the  sub-systems,  in  order  to  be  fully  compliant 
with  modularity  requirements:  a  major  advantage  gained  from  the  maximum  communality  between  these 
variants  is  the  possibility  to  control  different  combinations  with  the  same  GS,  reducing  the  formation  of 
operators. 

3)  The  external  interfaces  of  UAV  systems  are  also  concerned  by  standardisation,  in  order  to  meet  the 
growing  needs  of  inter-operability:  transit  of  data  by  a  satellite  network,  transmission  of  imagery  to 
other  services  for  intelligence  gathering  or  damage  assessment,  information  sharing  during  joint 
operations... 

Recommendations  of  NATO  through  the  Standardization  NATO  Agreements  or  STANAGs  are  the 
backbone  of  this  work  on  standardisation:  for  example,  STANAG  4609  on  Motion  Imagery  is  used  as  a 
basis  for  future  systems’  video  chain  design. 
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5.0  DATA  &  IMAGE  FUSION  FOR  THE  UAV  SYSTEM  OBSERVATION 
FUNCTION 

In  this  section  will  be  presented  a  wide  variety  of  process  grouped  under  the  same  denomination  of 
“fusion”.  What  we  call  here  “fusion”  is  a  process  which,  from  information  of  a  unique  or  several  kinds, 
obtained  from  a  unique  or  several  sources,  produces  either  an  improvement  of  input  information,  or 
information  of  a  new  kind. 

Fusion  algorithms  are  studied  now  for  a  long  time  in  various  fields  of  interest,  and  become  more  and  more 
attractive  with  the  exponential  growth  of  data  quantity  available  at  different  stages  of  the  decision  chain: 
with  the  continuous  development  of  data  link,  it  became  much  easier  to  transmit  a  large  amount  of  data 
rather  than  “filtrating”  it.  An  overwhelming  quantity  of  information  (data,  images,  sounds...)  can  be 
transmitted  to  the  final  stage  of  the  system,  very  often  a  human  operator,  which  has  to  perform  its  own 
real-time  processing,  at  the  major  risk  to  pick  up  the  “wrong”  information  when  submitted  to  a  critical 
operational  context.  In  this  case,  the  use  of  fusion  processes  at  different  levels  of  the  system  may  highly 
decrease  the  operator  workload. 

Recent  developments  in  electronics,  in  terms  of  computation  power,  memory  capacities,  transfer  rate  and 
compactness  made  realistic  to  integrate  such  processing  inside  embedded  airborne  payloads,  and  perform 
it  in  real  time  with  significant  results. 

Algorithms  and  combinations  presented  in  the  following  paragraphs  have  been  retained  after  examination 
of  several  quality  factors  listed  in  growing  order  of  importance:  ability  to  be  embedded,  robustness, 
operational  interest.  The  final  step  of  validation  of  these  functions  will  be  their  evaluation  in  operational 
conditions  by  expert  military  operators. 

Fusion  processes  are  grouped  according  to  the  system  level  at  which  they  can  be  applied:  at  the  sensor 
level,  between  different  OP  sensors,  between  OP  sensors  and  other  UAV  payloads,  between  OP 
information  and  data  kept  in  the  GS. 

5.1  Inside  UAV-OP  Sensor 

A  first  level  of  fusion  can  be  made  at  sensor  level,  by  processing  images  and  data  coming  from  a  same 
sensor,  to  add  a  functionality  to  the  sensor  or  to  achieve  significant  performance  increase.  These  process 
can  be  implemented  in  sensor  electronics  or  in  OP  electronics,  in  order  to  apply  the  same  processes  to 
different  imaging  sensors:  TI,  HDTV,  Spotter... 
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Process 

Process  principle 

Operational  interest 

Image 

stabilisation 

By  comparison  of  successive  images 
coming  from  the  same  sensor,  a  precise 
estimation  of  the  sensor  motion  between 
the  different  images  is  calculated,  allowing 
image  stabilisation  with  a  sub-pixel 
accuracy. 

•  Image  quality  improvement,  facilitating 
the  operator  work. 

•  Improvement  of  long  range  optronic 
performances  (Identification),  relying  on 
high  spatial  frequencies  in  the  image. 

Fine  detector 
calibration  and 
non-uniformity 
correction 

Motion  compensation  allows  to  compare 
different  pixels’  response  to  the  same 
thermal  or  luminous  sources  in  the  scene, 
making  possible  to  finely  compensate 
responsivity  variation  among  pixels  of  the 
same  detector. 

The  same  algorithm  allows  to  detect  bad  or 
“dead”  pixels. 

•  Image  quality  improvement. 

•  Improvement  of  optronic  performances 
relying  on  exploitation  of  pixel  size 
details. 

Automatic 
target  detection 
and  tracking 

After  stabilisation  and  detector  calibration, 
each  new  is  compared  with  the  preceding 
image  of  the  sequence  to  extract  the  map 
of  moving  or  varying  objects  in  the  image. 
Position,  speed  and  size  of  moving  objects 
are  filtered  at  lower  rate  to  perform  target 
selection  and  tracking. 

•  Reduction  of  operator  workload  during 
target  search  /  improvement  of  detection 
performances. 

•  Reduction  of  operator  workload  during 
target  tracking,  especially  for  multi¬ 
tracking. 

Image 

accumulation 
and  restoration 

Intensity  accumulation  applied  on  non¬ 
moving  pixels  drastically  increases  the 
Signal-to-Noise  Ratio  of  these  pixels, 
allowing  the  use  of  powerful  restoration 
algorithms. 

•  Improvement  of  optronic  performances, 

particularly  on  low  contrast  targets  (Long 
range  detection). 

Hyper¬ 

resolution 

An  over-sampled  image  of  the  scene  is 
obtained  by  high  precision  compensation 
of  the  sensor  motion  between  successive 
images,  with  sub-pixel  accuracy. 

•  Increase  of  image  resolution. 

•  Reduction  of  aliasing,  making  possible 
Wiener  restoration  on  images  initially 
produced  with  under-sampled  imagers. 

•  Improvement  of  optronic  performances 
relying  on  high  spatial  frequencies  in  the 
image  (Identification). 

5.2  Between  UAV-OP  Sensors  and  Equipments 

A  second  kind  of  fusion  processes  can  be  applied  between  different  sensors  of  the  observation  payload. 
These  processes  either  provide  an  additional  functionality,  or  increase  performances  of  another  processing. 
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Process 

Process  principle 

Operational  interest 

Image 

enrichment 

and 

annotation 

Data  produced  by  OP  are  collected  and 
processed  to  obtain  higher  level  data: 
Interesting  data  for  the  UAV  system 
operator  are  traduced  into  letters  and 
symbols,  overlaid  on  the  video  imagery. 

•  Reduction  of  operator  workload,  by 
presentation  of  complementary  data  directly 
on  the  image. 

Examples  of  sensors’  data:  LOS  orientation  and 
speed,  target  distance  measured  by  LRF,  position 
of  the  tracked  target. 

Example  of  secondary  level  data:  predicted  target 
direction  and  speed  obtained  by  processing  of  LRF 
target  range,  inertial  LOS  rate  measured  by 
gyrometers  and  tracking  errors. 

Multispectral 
image  fusion 

Video  flows  produced  by  two 
boresighted  imaging  sensors  (for 
example  a  thermal  imager  and  a  TV 
camera)  are  processed  to  produce  a 
single  image  combining  the  main 
features  visible  in  the  two  spectral  bands. 

The  algorithm  may  also  be  optimised  to 
enhance  the  readability  of  the  fused 
image. 

•  Increase  of  detection  capabilities:  the  fused 
image  shows  a  wider  spectrum  than  each 
single  image,  making  target  concealment  less 
efficient. 

•  Improvement  of  situational  awareness  by 

simultaneous  presentation  of  complementary 
information  sensed  in  different  spectral  bands. 

•  Reduction  of  the  operator  workload:  for 

surveillance  missions,  the  operator  watches  a 
single  display,  without  switching  time  and 
eye-brain  adaptation  time. 

•  Technical  interest:  a  single  data  link  transmits 

information  requiring  two  links  without 

fusion. 

Image 

stabilisation 

Line-Of-Sight  residual  motion  is 
measured  by  inertial  sensors 

(gyrometers,  accelerometers)  integrated 
in  the  mechanical  structure  of  the  Sensor 
Pack,  and  filtered  to  perform  image 
motion  compensation. 

•  Same  operational  advantages  as  in  3. 1 . 

Laser- 

enhanced 

Imagery 

1)  A  laser  is  synchronised  with  a  low- 
light  level  TV  camera  to  increase 
target  illumination  during  night-time 
operations. 

2)  Range  gated  laser  imagery:  A  laser  is 
used  as  target  illuminator  and  range 
finder,  precisely  synchronised  with 
an  imaging  sensor  to  eliminate  image 
blur  due  to  light  scattering  and 
reflection  along  flight  path. 

•  Increase  of  optronic  performances  (D,  R,  I) 
during  normal  weather  conditions. 

•  Enlargement  of  operational  domain,  by 
compensation  of  degradation  induced  by  bad 
weather  conditions  (snow,  fog). 
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5.3  Between  UAV-OP  and  Other  UAV  Modules 

Onboard  the  UAV,  the  OP  is  placed  in  a  very  rich  data  environment,  produced  by  various  other  payloads: 
Inertial  Navigation  System,  Global  Positioning  System,  Radar,  SAR. . . 


Process 

Process  principle 

Operational  interest 

Geo- 

Referencing 

UAV  position  and  orientation  are 
combined  with  angular  measurements  of 
the  gyrostabilized  platform  to  compute 
LOS  orientation  in  the  terrestrial 
reference  in  real-time.  The  use  of  a  LRF 
to  measure  Surface-Sight  Point  range 
allows  to  know  target  position  in  3D. 

After  compensation  of  optical  effects, 
each  pixel  of  the  image  is  linked  to  its 
position  on  the  earth  sphere. 

•  Without  LRF:  Medium  precision,  passive 
target  localisation. 

•  With  LRF:  more  precise,  but  less  discrete 
Surface-Sight  Point  localisation. 

Geo-Tracking 

Thanks  to  INS  /  GPS  information,  UAV 
motions  are  compensated  to  slave  the 
LOS  on  a  target  defined  by  its 
coordinates  in  the  terrestrial  reference 
(Latitude  &  Longitude). 

These  coordinates  may  be  defined  by  the 
operator  or  sent  by  another  payload 
(radar). 

•  Reduction  of  the  operator  workload:  the 
operator  relies  on  the  automatic  LOS 
slaving  managed  by  the  OP,  and  fully 
concentrates  on  its  value-added  activity: 
images  and  data  analysis  and  exploitation. 

Geo-Scanning 

The  same  algorithms  are  used  to  slave  the 
LOS  along  sweeping  or  scanning 
patterns,  to  ensure  a  full  coverage  of  a 
area  of  interest  on  the  ground,  whatever 
the  linear  and  angular  motions  of  the 
UAV. 

Image  depth 
mapping 

Combination  of  pixel  motion 

measurement  techniques  (e.g.  optical 
flow)  with  data  provided  by  INS/GPS 
allows  to  obtain  distance  estimation  of 
objects  in  the  image. 

Operational  interest  varies  following  the  types 
of  UAV: 

•  On  tactical  UAVs,  assistance  during 
navigation  in  low-level  flights. 

•  On  mini-UAVs,  participation  to  the 
vehicle  automatic  piloting  function,  by 
altitude  and  attitude  estimation. 

5.4  Between  UAV-OP  and  GS  DATA 

Additional  synergies  can  be  obtained  by  combination  of  data  produced  by  embedded  payloads  and 
information  available  on  the  ground  station.  These  processes  are  applied  at  UAV  system  level. 
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Process 

Process  principle 

Operational  interest 

Image  Geo- 
Referencing  with 
alignment  on 
Digital  Terrain 
Elevation  Data 
and  synthetic 
images 

Main  features  extracted  from  the  OP  images 
are  correlated  with  a  3D  model  of  the  interest 
area,  elaborated  during  the  mission 
preparation. 

A  range  measurement  thanks  to  the  OP  LRF 
provides  additional  improvement  of  image-to- 
model  fitting. 

•  High  precision  3D  geo-referencing 
of  each  pixel  of  the  image. 

Automatic  Target 
DRI  and 
localisation 

Objects  “silhouettes”  are  extracted  from  the 
image  by  segmentation,  and  correlated  with  a 
large  database  constituted  by  precise  3D 
modelling  of  possible  targets:  planes, 
helicopters,  tanks,  shelters... 

•  Automatisation  of  DRI  tasks  allows 
the  processing  of  very  large  amount 
of  information. 

6.0  PROPOSITION  OF  UAV  OBSERVATION  FUNCTION  ARCHITECTURE 

6.1  Architecture  Design  Work 

Main  requirements  collected  in  the  preceding  steps  of  the  study  and  presented  in  previous  sections  are 
summarised  in  the  following  list: 

•  Compatibility  of  proposed  OP  architecture  with  existing  UAV  system  architecture, 

•  Multisensor  Observation  Payload:  Integration  of  several  sensors  of  different  types, 

•  Sensor  Modularity:  Multi  -  configuration  OP  payload  with  identical  external  interfaces, 

•  High  level  of  autonomy  and  automation  of  the  OP, 

•  High  operational  performances:  DRI,  Geo-localisation, 

•  Multi-carrier  compatibility, 

•  UAV  system  inter-operability, 

•  Integration  at  sensor  and  OP  level  of  data  and  image  processes  bringing  operational  added  value, 

•  Transmission  by  the  OP  of  information  with  sufficient  accuracy  to  allow  the  application  of  fusion 
processes  at  upper  system  levels. 

Basing  on  these  data,  we  propose  a  UAV  observation  function  architecture  designed  to  allow  a  maximum 
use  of  data  and  image  fusion  techniques,  at  different  levels  of  the  UAV  system.  The  work  to  build  this 
architecture  followed  two  main  axes: 

•  Functionalities  repartition  between  the  different  elements  of  the  imagery  chain; 

•  Definition  of  interfaces  and  data  links  between  these  different  modules. 

The  repartition  of  functions  relied  on  the  following  rules: 

•  Superposition  of  functional  and  hardware  architectures, 

•  Lower-level  functions  management  is  left  to  lower-level  sub-systems,  in  order  to  reduce  control 
loops  latency  and  volume  of  exchanged  data, 

•  Maximum  communality  of  high-level  functions  management  to  avoid  useless  redundancies. 
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These  rules  result  in  repartition  of  management  capabilities  and  thus  in  increase  of  electronic  volume 
among  the  different  levels  of  the  OP  architecture;  their  application  may  be  limited  in  function  of  available 
volume. 

The  definition  of  interfaces  and  data  links  was  driven  by  these  main  principles: 

•  Maximization  of  data  links  capabilities  (rate,  bit  depth)  to  allow  the  most  accurate  processing  after 
data  transmission  to  higher  system  level, 

•  Choice  of  internal  OP  interfaces  to  reach  maximum  OP  re-configurability  and  sensor  modularity, 

•  Standardization  of  external  OP  interfaces  to  reach  maximum  OP  /  UAV  interchangeability 
(a  given  OP  on  different  UAV  types,  or  different  OP  on  a  given  UAV). 

6.2  Resulting  Architecture 

The  proposed  architecture  is  summed  up  by  the  following  figure: 
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Main  characteristics  of  this  architecture  are  the  following: 

Electronics  of  each  sensor  is  in  charge  of  sensor  low-level  functions  management,  and  specific  data  and 
image  processing,  in  order  to  achieve  sensor  modularity  at  OP  level,  and  to  avoid  overload  of  OP 
electronics  (Example  of  TI  specific  image  processing:  Non-Uniformity  correction). 

Video  and  data  links  inside  the  OP  (between  OP  sub-assemblies  and  OP  electronics)  are  dimensioned  to 
achieve  the  highest  possible  performances  with  processes  implemented  inside  OP  Management  and 
Processing  electronics;  Standardisation  of  these  links  is  a  secondary  priority,  compared  to  data  link  rate, 
bandwidth  and  bit  depth. 

OP  Management  and  Processing  electronics  is  in  charge  of  functionalities  requiring  multi-sensor  /  multi¬ 
payload  information  processing,  with  very  short  control  loops  (for  example  Tracking,  Geo-Tracking...), 
and  of  processes  which  can  be  equally  applied  on  different  sensors:  image  stabilisation,  hyper-resolution. 

OP  electronics  also  assumes  responsibility  of  OP  to  UAV  interfaces: 

•  For  commands  and  data  exchange,  OP  is  connected  to  UAV  MCU  and  other  UAV  payloads 
thanks  to  the  UAV  Mission  Bus,  defined  at  UAV  system  level. 

•  For  video  link,  standardisation  is  the  first  priority  to  comply  with  inter-operability  requirement, 
prior  to  video  link  technical  characteristics:  For  example,  STANAG  4609  on  Motion  Imagery 
recommends  10  bits  digital  video  formats,  whereas  IR  sensors  usually  produce  images  with  more 
than  12  bits  precision.  This  leads  to  locate  in  the  OP  Management  and  Processing  electronics 
processes  requiring  very  high  image  precision:  for  example,  automatic  target  detection  &  target 
fusion. 

Image  and  data  pre-processing  before  transmission  between  UAV  and  GS  are  taken  in  charge  by  a  specific 
module  of  the  UAV  System,  because  they  are  strongly  related  to  the  type  and  characteristics  of 
transmission. 

The  choice  of  standardised  digital  data  &  video  links  allows  further  diffusion  of  information  without 
additional  degradation  due  to  format  conversion,  which  is  necessary  to  perform  supplementary  fusion 
processes  at  higher  levels  of  the  system. 
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Introduction 


★  SAGEM  :  European  leader  for  UAV 

O  Main  supplier  for  Tactical  UAV  systems  in  Europe 
O  Observation  Payload  (OP)  for  Medium  Altitude,  Long  Endurance  UAV 
O  Proposition  of  platoon-level  mini-UAV 

★SAGEM  UAV  success  based  on  long-term  expertise  in  : 

O  Optronic  systems  (European  n°1) 

O  Inertial  Navigation  Systems  (European  nQ1,  World  n°3) 

O  Global  Positioning  Systems 
O  RF  communications  and  Data  Links 

★  Development  of  next-generation  UAV  observation  payloads 

O  Specific  study  to  maximise  synergies  thanks  to  fusion  techniques 


e 

Branche  Defense  et  Securite 


Synthesis  of  the  study 


UAV  System  R&D  Unit 
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—  SAGEM  airborne  optronic  payloads 

★  SAGEM  :  European  leader  for  gyrostabilised  observation  payloads, 
installed  on  every  type  of  airborne  carrier 


O  Multi-role  Helicopters 


O Patrol  Aircrafts 


OTactical  UAV 
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Branche Defense etsecunte  q  More  than  300  Gyrostabilised  Payloads  in  operational  service 


OTransport  Helicopters 
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-SAGEM  experience  in  gyrostabiiised  payloads  (1) 

★  Evolution  of  sensors  integrated  in  airborne  OP 


★  SAGEM  IRIS 


O  Gen,  II  Thermal  Imager 
O  8-12  |jm  spectral  band 
★  TV  camera 


★ 

★ 


Infrared  Line  Scanner 
8-12  pm  spectral  band 
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SAGEM  MATIS 
OGen.  Ill  Thermal  Imager 
03-5  pm  spectral  band 
High  definition  TV  camera 
Laser  Range  Finder 
Laser  Designator 
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-SAGEM  experience  in  gyrostabilised  payloads  (2) 

★  Constant  evolution  of  functional  area  covered  by  OP 


★  1991  :  PGS 

★  2-axis  stabilisation 

★  1  completely 
independent  Sensor 
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★  2001  :  OLOSP410 

★  4-axis  stab. 

★  Multi-sensor  configuration 

★  Integrated  management  electronics 


★  1995  :  OLOSP  350 

★  4-axis  stab. 

★  2  sensors  :  IR  +  TV 


★  OLOSP  NG 

★  High  accuracy  stab. 

★  Modular  architecture 

★  Advanced  functionalities 


UAV  Observation  Function 


★  Observation  function  performance  relies  on  every  element 
of  the  Imagery  chain 
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Imagery  Chain  :  blue 


Ground  Station  (GS): 

•  Video  &  Data  Reception 

•  Video  &  Data  Processing 

•  Man-Machine  Interface 


Specificity  of  UAV  context 
for  observation  payloads 


★  imagery  and  data  exploited  on  the  ground  after  transmission 

O  Specific  processing  to  reduce  loss  of  quality  due  to  transmission 

★  Higher  level  of  automation 

O  Necessary  for  mission  permanency  during  possible  transmission  loss 

O  Reduction  of  exchanged  data  to  high-level  information  to  comply  with 
data  link  limitation 

★  Unmatched  operational  &  technical  requirements 

O  Optronic  performances  (D,  R,  I) 

O  Contribution  to  Geo-localisation 
O  Miniaturisation 

★  integration  in  higher-level  information  systems 

O  Inter-operability  requirement 
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Specificity  of  UAV  context 


★  Constant  diversification 
of  UAV  systems 


CRECERELLE- 1993 
Tactical  UAV 
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Bell  -  Rheinmetall  -  SAGEM  cooperation 

★  Interchangeability  of  UAV  and  OP 

★  Definition  of  a  scalable  architecture 
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-  Data  &  Image  Fusion  techniques 

★  Fusion  :  process  producing  significant  synergy  between 
information  produced  by  sensors  or  other  data  sources 


★  Growing  interest  for  fusion  techniques 

O  Increasing  volume  of  available  data 
O  Requirement  for  reduction  of  operator  workload 
O  Development  of  embedded  electronics 

★Applicable  at  different  levels  of  the  UAV  observation  function 
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Data  &  (mage  Fusion  techniques 


★  inside  sensors:  fusion  of  successive  images 

O  Image  stabilisation  by  image  processing 
O  Fine  detector  calibration  &  non-uniformity  correction 
O  Automatic  target  detection 
O  Hyper-resolution 


★  inside  OP:  between  different  image  &  data  sources 


O  Automatic  target  tracking 
O  Image  enrichment  and  annotation 

O  multi-spectral  image  fusion 
O  Laser  enhanced  imagery 


Data  &  (mage  Fusion  techniques 


★  Between  OP  and  other  payloads  of  the  same  aircraft: 
INS,  GPS,  radar... 

O  Pixel  Geo-referencing 

O  Geo-Tracking:  automatic  LOS  orientation  in  terrestrial  reference 
O  Geo-Scanning:  automatic  area  scanning  in  terrestrial  reference 
O  Passive  image  depth  mapping 

★  Between  OP  and  GS  information  (target  database...) 

O  Image  Geo-Referencing  with  alignment  on  Digital  Terrain 
Elevation  Data  and  synthetic  images 

O  Automatic  Target  DRI  and  localisation 
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Example  :  Multi-sensor  image  fusion 


★  Multi-spectral  image  fusion  on  OLOSP  410 

O  SAGEM  IRIS  Thermal  Imager  (8-12  |jm) 

O  Near- Infra  red  TV  camera  (0.6  -  0.9  pm) 

★Operational  interests 

O  Increased  detection  capabilities 

O  Improvement  of  situational  awareness:  Reduction  of  the 
operator  workload  during  search  task 

O  A  single  data  link  transmits  images  of  two  sensors 


★  Choice  of  multi-level  Laplacian  pyramid  algorithm 


O  Robust  and  efficient  algorithm 
O  Well  fitted  to  FPGA  used  in  embedded  electronics 
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— Multi-spectral  fusion  with  Laplacian  pyramid 

★  Images  are  pre-processed  before  fusion 


IR  video 
flow 


TV  video 
flow 


IR  Image  distortion 
correction 


TV  Image  distortion 
correction 


Fusion  operation 

IR  /  TV  image 

Decomposition 

boresighting 

Decision 

<=> 

Recombination 

Fused 
video  flow 


O IR  and  TV  images  decomposition  on  Laplacian  pyramid  (l,Rk  &  lTvk) 


O  Application  of  decision  mask 
on  each  pyramid  level : 


: 


>  a 


elsewhere 
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O  Fused  image  formation  by  recombination  of  fused  images  (Dk)  pyramid 
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Multi-sensor  image  fusion  trials  : 
Airfield  approach  during  night  flight 


Observation  payload  design 


★  Identification  of 

prioritary  requirements 


High  optronic  performances 


Maximum  robustness 


Sensor  modularity  ; 
Multi-sensor  capability 


★  Design  rules 

O  Functionalities  repartition 
among  imagery  chain 
elements 


Superposition  of  functional 
and  hardware  architectures 


Repartition  of  management 
capabilities 


High  level  of  automation 


Multi-carrier  compatibility 


Maximum  operational 
benefits  from  fusion 
techniques 


Compactness 


O  OP  Interfaces  definition 

Maximum  standardization 
of  interfaces 


Maximum  data  link 
capabilities 


O  Minimization  of 
mass  &  volume 


★  Major  design  choices 

O  Specific  processing 
implemented  in  sensor 
electronics 


O  Full  precision,  high 
bandwidth  internal  data 
links 


O  OP  electronics 
dimensioned  for : 

>  OP  management 

>  Data  &  image 
processing 

O  Standardization  of 
external  interfaces  : 

>  STANAG  4609  for 
motion  imagery... 
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Imagery  chain  architecture 


UAV 


GPS 


INS 


UAV 

Management 
&  Control 
Unit 


Observation  Payload  (OP) 


i 

X 


Sensor  Pack  (SP) 


Imagery  Chain  :  blue  elements 


Air 

Vehicle 


Other  information  systems  : 
Headquarters... 


Other 

Payloads 

(radar...) 


.  Video  & 
Data  Link  *** 
Payload 


OP 

Management 

& 

Processing 

Electronics 


Imaging  sensor  1 


Imaging  sensor  2 


Laser  sensor  1 


Laser  sensor  2 


Angular  &  inertial 
sensors 


Gyro-Stabilized  Platform  (GSP) 


- 

- 


GS  Video  &  Data  Link  Module 


GS  Man-Machine 
Interface 
(MMI) 


j  Ground  Station  (GS) 


A 

▼  * 


GS  Video  &  Data 
Processing  Unit 


3D  geographic  model 
&  Target  database 


Full  precision,  high  bandwidth  data  link 

— ► 

and  video  link 

— ► 

Standardized  data  link 

◄ . ► 

and  video  link 

. ► 

Compressed  data  link 

◄ - ► 

and  video  link 

- -► 

Conclusion 


★Application  of  fusion  techniques  results  in  major 
improvements  of  UAV  Observation  Function 

O  Enhanced  optronic  performances  (Detection,  Reconnaissance, 
Identification)  &  geo-localisation 

O  Reduction  of  the  operator  workload 


★  Observation  Payload  design  allows  maximum  use  of 
fusion  techniques  @  every  system  level 
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—  Data  &  Image  Fusion 
for  multisensor  UAV  Payloads 
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